Testing Strategies for Model-Based Development

This report presents an approach for testing artifacts generated in a model-based development process. This approach divides the traditional testing process into two parts: requirements-based testing (validation testing) which determines whether the model implements the high-level requirements and model-based testing (conformance testing) which determines whether the code generated from a model is behaviorally equivalent to the model. The goals of the two processes differ significantly and this report explores suitable testing metrics and automation strategies for each. To support requirements-based testing, we define novel objective requirements coverage metrics similar to existing specification and code coverage metrics. For model-based testing, we briefly describe automation strategies and examine the fault-finding capability of different structural coverage metrics using tests automatically generated from the model

Model-Based Safety Analysis

System safety analysis techniques are well established and are used extensively during the design of safety-critical systems. Despite this, most of the techniques are highly subjective and dependent on the skill of the practitioner. Since these analyses are usually based on an informal system model, it is unlikely that they will be complete, consistent, and error free. In fact, the lack of precise models of the system architecture and its failure modes often forces the safety analysts to devote much of their effort to gathering architectural details about the system behavior from several sources and embedding this information in the safety artifacts such as the fault trees. This report describes Model-Based Safety Analysis, an approach in which the system and safety engineers share a common system model created using a model-based development process. By extending the system model with a fault model as well as relevant portions of the physical system to be controlled, automated support can be provided for much of the safety analysis. We believe that by using a common model for both system and safety engineering and automating parts of the safety analysis, we can both reduce the cost and improve the quality of the safety analysis. Here we present our vision of model-based safety analysis and discuss the advantages and challenges in making this approach practical

Safety and Software Intensive Systems: Challenges Old and New

There is an increased use of software in safety-critical systems; a trend that is likely to continue in the future. Although traditional system safety techniques are applicable to software intensive systems, there are new challenges emerging. In this report we will address four issues we believe will pose challenges in the future. First, the nature of safety is continuing to be widely misunderstood and known system safety techniques are not applied. Second, our ability to demonstrate (certify) that safety requirements have been met is inadequate. Third, modeling and automated tools, for example, code generation and automated testing, are introduced in a hope to increase productivity; this reliance on tools rather than people, however, introduces new and poorly understood problems. Finally, safety-critical systems are increasingly relying on data (configuration data or databases), incorrect data could have catastrophic and widespread consequences

Structuring Formal Requirements Specifications for Reuse and Product Families

In this project we have investigated how formal specifications should be structured to allow for requirements reuse, product family engineering, and ease of requirements change, The contributions of this work include (1) a requirements specification methodology specifically targeted for critical avionics applications, (2) guidelines for how to structure state-based specifications to facilitate ease of change and reuse, and (3) examples from the avionics domain demonstrating the proposed approach

Let’s Not Forget Validation

Abstract — As we are moving from a traditional software development process to a new development paradigm where the process it largely driven by tools and automation, new challenges for verification and validation (V&V) emerge. Productivity improvements will in this new paradigm be achieved through reduced emphasis on testing of implementations, increased reliance on automated analysis tools applied in the specification domain, verifiably correct generation of sourcecode, and verifiably correct compilation. The V&V effort will now be largely focused on assuring that the formal specifications are correct and that the tools are trustworthy so that we can rely on the results of the analysis and code generation without extensive additional testing of the resulting implementation. Traditionally, most research efforts have been devoted to the verification problem. In this position paper we point out the importance of validation and argue that if we fail to adequately address the validation problem, the impact of verifying code generation and compilation will be limited. I

Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing

Model checking techniques can be successfully employed as a test case generation technique to generate tests from formal models. The number of tests cases produced, however, is typically large for complex coverage criteria such as MCDC. Test-suite reduction can provide us with a smaller set of test cases that preserve the original coverage—often a dramatically smaller set. One potential drawback with testsuite reduction is that this might affect the quality of the testsuite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing formal models of software, we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results show that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications